The Calculus of Jacobian Adaptation

For many problems, the correct behavior of a model depends not only on its input-output mapping but also on properties of its Jacobian matrix, the matrix of partial derivatives of the model’s outputs with respect to its inputs. This paper introduces the J-prop algorithm, an efficient general method for computing the exact partial derivatives of a variety of simple functions of the Jacobian of a model with respect to its free parameters. The algorithm applies to any parametrized doublydifferentiable data-driven feedforward model, including nonlinear regression, multilayer perceptrons, and radial basis function networks. J-prop has as special cases some earlier algorithms, such as Tangent Prop and Double Backpropagation, but it can also be used to optimize the eigenvectors, eigenvalues, or determinant of the Jacobian, making it quite general with respect to the model types, applications, and objective functions that can be used. Applications of J-prop include forcing stability and sensitivity conditions, regularization, building low-complexity encoder-decoder networks, exploiting prior knowledge, designing controllers, and blind source separation.